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ABSTRACT 

Two studies are reported. In the first, the 
effectiveness of student ratings in improving college teaching was 
investigated. Teachers in five diverse colleges were assigned 
randomly to a feedback (treatment) , no^f eedback (control) , or 
posttest group. The feedback and no-feedback groups used a student 
rating from in one of their classes at midsemester and responded to a 
self-evaluation form. The feedback group received a summary of 
student responses. Both groups administered the form again at the end 
of the semester, as did the posttest group. If student feedback 
improved instruction, end-of-semester ratings of the feedback group 
should have been better than the other groups. But multivariate 
analysis of variance results indicated no significant differences 
among the groups. Regression analyses indicated that student feedback 
did effect changes in teachers who had rated themselves more 
favorably than their students had rated them. In the second study, 
teacher self- ratings were compared to student ratings to investigate 
the extent to which instructors learn something from students about 
their teaching. Teachers generally evaluated their teaching somewhat 
differently from the way their students evaluated it. Comparisons 
across items between student and faculty responses indicate that 
instructors are aware of many of their strengths and weaknesses but 
see themselves more favorably in absolute terms. (Author/KM) 
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Preface 



This report is presented in two parts. In the first part, the effective- 
ness of student ratings in improving college teaching :.s investigated. An 
experimental study was conducted at five diverse colleges to investigate the 
question: To what extent do college teachers alter their instructional practices 
after receiving feedback from students? The "treatment" in this study purposely 
paralleled what is typically done at most colleges that use student ratings— 
that is, the results were seen only by the individual teacher, and the Inter- 
pretation of the ratings as well as what to do about them were left vp to the 
teacher . 

In the second part of this report, teacher self-ratings are compared 
to student ratings in order to investigate the extent to which instructors 
learn something from students about the?r teaching. Because this second part 
will eventually be published separately descriptions of the instruments, 
sample, etc. are repeated. 

Thanks are due to the teachers and students who participated, and in 
particular to the staff member at each college who helped coordinate the study. 
Also, to Charles Har and Donald Rubin for advice with statistical analyses-, 
to Gordon Davis and Dorothy Thayer for computer programming, and to Marian Helms 
for unerring attention to the many deadlines and details. 

John A. Centra 
Research Psychologist 



I. THE EFFECTIVENESS OF STUDENT FEEDBACK IN 
MODIFYING COLLEGE INSTRUCTION 



Formal student evaluations of courses and teaching have been receiving a 
good deal of attention and use at many colleges and universities. The results 
of these evaluations most often are seen cily by instructors and are intended 
to help improve their teaching. Underlying this intended use is the assumption 
that the instructors will use the information to alter and improve their 
teaching. It is an assumption open to question. 

Presumably, instructors value student opinion enough to change their 
teaching behavior when it is evaluated less favorably than the instructors 
might desire or expect. The theoretical justification for this belief, as 
developed by Gage, Runkel, and Chatterjee (1963) and by Daw and Gage (1967), 
may be found in equilibrium theory. Accordingly, when student feedback 
creates a condition of imbalance (Heider, 1958), asymmetry (Newcomb, 1959), 
or dissonance (Festinger, 1957) in an instructor, one might expect the 
instructor to change in the direction desired by students in order to re- 
store a condition of "equilibrium," Following a suitable lapse of time, 
such changes should be reflected in a second description of teacher behavior. 
There is some evidence that student feedback does indeed have a positive 
effect on teaching performance, although the evidence is far from conclusive, 
particularly at the college level, Tuckman and Oliver (1968), using 286 
teachers of vocational subjects in high school and technical institutes, 
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found that instructors who received student feedback showed greater "gains 11 
in student ratings, as measured by changes in students 1 ratings after a 
12-week interval, than instructors who received no feedback (actually all the 
change scores were negative, with positive changes or "gains 11 being simpJ/ * 
less of a negative score). Changes in ratings of teaching were also reported 
by Bryan (1963), using teachers of academic subjects at the secondary level, 
and by Gage, Runkel, and Chatterjee (1963), who experimented with sixth-grade 
teachers . 

The results at the college level, however, have thus far been less positive. 
Miller (1971) reported that end-of-semester student ratings for teaching assis- 
tants who had received midsemester feedback did not differ from end-of-semester 
ratings for teaching assistants who did not receive the feedback. But because 
of the small and limited nature of the sample (36 teaching assistants assigned 
to discussion sections in three courses), the results of the Miller study are 
very tentative. 

The preceding studies neglected to include a number of relevant variables 
that might be expected to be related to changes in teaching. None of the 
studies investigated the instructor's awareness of his own teaching practices 
as indicated by self-ratings. On the basis of equilibrium theory, one could 
hypothesize that the greater the gap between student ratings and faculty 
self-ratings, the greater the likelinood that there would be change in 
instruction, since large differences would create the greatest amount of 
imbalance or dissonance in instructors. None of the preceding studies, 
furthermore, looked at possible variations in changes across subject areas 
nor did the} investigate the sex of the instructor as still another variable. 
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Finally, the number of years of teaching experience is a particularly critical 
variable which was included in only one of the preceding studies. In that 
study (Tuckman & Oliver, I968), the expectation that less experienced t.eachers 
would be more likely to change was not supported by the results. 

The primary purpose of the present study was to investigate in some depth 
the effects of student feedback on teaching at the college level. Included as 
variables in the study were the instructor's sex, teaching experience, and self- 
ratings, as well as the subject area of the course. These instructor or course 
characteristics were examined to determine their relationship to instructional 
changes. The study was carried out at several types of post-secondary institu- 
tions in order to investigate also the possibility that changes occur at some 
colleges but not at others, 

A second purpose of the study, presented as a second part rf this report, 
was to learn more about teacher self-ratings and, in particular, their rela- 
tionship with ratings provided by students. To what extent do instructors 
learn something from students about their teaching? Comparisons between 
student and teacher ratings over a wide range of instructional practices might 
underscore the need for student feedback, as well as highlight specific areas 
of instruction where feedback is most essential. 

Method 

The Sample of Institutions 

Five colleges which did not have a formal program of student ratings of 
instruction participated in the study. These were among 50 to 60 colleges 
contacted in the late spring of 1971 with a brief letter describing the 
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project (see Appendix A). Each of the institutions was asked to return a pre- 
printed postcard vith their replies to three questions: 

1. Does your institution have a formal program of student 
ratings of faculty and/or courses (either student conducted 
or institution sponsored)? 

2. In the absence of an institutionalized program, do mc:~\ of the 
faculty obtain student ratings of their own? 

3. Will your institution be on the semester system next fall? 

Only colleges which did not have a formal student rating program, whose 
faculty generally did not obtain ratings of their own, and which were on a 
semester system were considered for the study. At these institutions the fac- 
ulty would not be familiar with the way students viewed their instruction, and 
the formal feedback might therefore result in changes in teaching practices. 
Because the initial set of ratings was to be collected at midsemester (and 
the second set at the end of the semester), it was also important that col- 
leges be on a semester system rather than on the shorter quarter system, 
thereby enabling a longer period of time for changes to occur. 3y collecting 
both sets of ratings within one semester, moreover, the same students and the 
same courses could be [used. 

Nine the colleges that met the above specifications were invited to 
participate in the study, with the hope that five or six would be able to do 
so. The five that did accept were located in three states and represented 
several types of colleges, including two state colleges (one of which had a 
predominantly black enrollment), a liberal arts college, a multipurpose college, 
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and an urban community college. This diversity among the particpating col- 
leges provided a wide context of teaching environments. 

Procedure 

All teaching faculty from four of the five institutions were asked to 
participate in the first phase of the study. At the fifth institution, which 
was also the largest, all but 30 members of the faculty were invited to 
participate in the study. Those 30, chosen at random, were subsequently asked 
to participate in the study at the end of the second semester. 

In only one of the four colleges was the faculty told the full details of 
the study and, in particular, that student feedback would be purposely withheld 
from some of them. At the other four colleges the faculty were told that the 
project was "investigating what students are able to evaluate in the classroom 
and how useful this information might be to the individual instructor . 11 (Se? 
Appendix A for examples of letters; in one instance the writer spoke at a 
faculty meeting in general terms about the project.) 

Faculty members were assured that only they would see their individual 
rating reports. This assurance undoubtedly contributed to the excellent 
cooperation from the faculties; in fact, between 70-90$ of those at each 
institution participated in one or more phases of the study. 

Teache.s within each department of each institution were randomly assigned 
to one of three groups : 

1. the feedback group, which administered a rating' form at midsemester 
and received a summary of results (feedback) within a week; this 
group also received comparative data hased on responses in 75 classes 
at a sixth institution which had tried out several of the items 
during the Spring 1971 term (see Appendix B) ; 

ERiC 



2. the no- feedback group, for which student ratings were collected but 
withheld at midsemester; and 

3* the "posttest" group, which used the rating form only at the end of 
the semester in order to determine whether the midsemester ratings 
had a sensitizing effect on student raters or teachers. 

Each teacher was asked to use the questionnaire in one class of their 
choice. End-of-semester as well as midsemester ratings were collected for 
both the feedback and no- feedback groups. Both midsemester and end-of-semester 
ratings were collected during the fall semester of 1971. A single semester 
instead of two successive semesters was used initially in order to enable the 
same students to provide both sets of ratings in the same course. Moreover, 
a suggestion that has sometimes been made is that instructors should obtain 
feedback from students at midsemester so that those students who provided 
the information might benefit. 

In Table 1 the number of teachers participating in zhe study at mid- 
semester and at the end of the semester are listed by college and by group. 
As might be expected, some of the teachers who used the form at midsemester 
did not remain in the stuaj for the critical end-of-semester administration. 
The question then is whether those who dropped out of the study after using 
the form at midsemester biased the final sample. Were the dropouts, for 
example, generally the more poorly rated teachers? To examine this question, 
comparisons were made between three sets of scores: teachers who dropped out 
vs. those who stayed in for group 1, the same comparisons for group 2, and 
teachers who stayed in for both group 1 and group 2- Thus in-drop comparisons 
within the two groups and in-in comparisons across the groups were made for 
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Table 1 

Number of Teachers Participating in the Study at Midsemester 
and at End of Semester, by College and Group 

Group 1 Group 2 Group 3 

Feedback No Feedback Posttest 



College 


Mid- 
Semester 


End of 
Semester 


Mid- 
Semester 


End of 
Semester 


End of 
Semester Only 


1 


26 


22 


25 


22 


25 


2 


50 


35 


55 


1*5 


3h 


3 


33 


21 


32 


23 


19 


It 


h9 


h2 


52 


k8 


h5 


5 


19 


17 


2k 


21 


17 


Totals 


177 


137 


188 ' 


159 


1U0 
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the 23 items used at midsemester , Out of 69 tests of significance (3 x 23), 
differences were statistically significant (,05 level) for four of the items. 
Since differences for that many items could be expected tc * w .i the basis 

of chance (at the 5 per cent level of significance), it is safe to conclude 
that the teachers who dropped out of the stucy were very much like those who 
continued (at least in ratings given by students) and, equally important, 
thac the feedback and no-feedback groups were very similar in their student 
ratings at midsemester. 

Instruments 

Student ratings or descriptions of instruction were measure at mid- 
semester by a 23-item "Student Instructional Report" (SIR) , Induced were 
items that faculty members in an earlier study had identified as providing 
information they would like from students, 1 In particular, items that 
reflected instructional procedures or behavior that teachers presumably 
could change wer^ used in the study. Among the areas included were course 
objectives, instructor preparation and organization, student-faculty inter- 
action, student effort, and course difficulty and scope. 

The end-of-semester SIR contained the same 23 items; the items were 
slightly rearranged and presented on a machine-scoreable answer sheet. In 
addition, it contained several additional items eliciting overall rather 
than specific ratings; since most of the teachers would be administering the 

lf The questionnaire was developed the previous year with the cooperation 
of Northwestern University faculty and students. Further information on its 
development may be found in Centra (1972). 



items foi ae jeconJ time in the same course, it was hoped that the additional 
items vould encourage its repeated use, A copy of the midsemester and end-of- 
semeste'r questionnaires may be found in Appendix C. 

Item responses for 19 of the items were on a four-point agree-disagree 
scale. The remaining four items employed a four- or five-point scale with 
varying responses. Each instructor received at the appropriate time a summary 
report that included the mean and standard deviation for each item and the 
percentage of students in the class that gave each response . As mentioned 
earlier, the feedback group also received with their midsemester results 
comparison data from 75 instructors who had administered SIR at the end of the 
Spring 1971 semester (Appendix B). 

Item reliabili ties. One way tc estimate the reliability of each item 
is through analysis of variance, a method that is somewhat similar to averag- 
ing all possible split-half correlations among student responses. By comparing 
the variance of responses within classes to the variance between classes, an 
e. tinted reliability coefficient may be obtained (see Winer, 1962, pp. 12*1-132, 
for additional details). A ]ow reliability estimate, according to this method, 
would suggest a great deal of variance in responses among students within a 
typical class, or very Little variation in the average responses over a sample 
of instructors, or both. Ideally it is desirable to have just the opposite — 
more differentiation between instructors than between the student responses in 
each class. Through use of the Spearman-Brown fornula, estimated reliabilities 
for classes of various sizes may then be computed. 

Analysis of variance reliability estimates for the SIR items were generally 
above .70 for 20 or more students in a class and slightly less for 15 or more 
students, ^hese values would seem to be quite acceptable. The data are 
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presented in Appendix D. The lowest reliabilities may be noted on items 
dealing with such aspects as the level of difficulty of the coarse, for 
which students within a class might more likely be expected to differ, 

Test-ret est is another frequently employed type of reliability. The 
correlations between midsemester and end-of-semester responses within the 
no- feedback and feedback groups separately could be considered as a form of 
test-retest reliability. Those correlations, as indicated in Appendix E, were 
moderately high, with the majority of items near or above ,70, While a shorter 
time lag would likely have resulted in higher stability of responses within 
each group, it is important to keep in mind that good stability with each of 
the groups does not ^legate the possibility of feedback effects, i,e., of 
finding differences between the two groups. Instructors within each group 
could be ordered or ranked similarly at both midsemester and end-of-semester 
but the mean difference between the groups could be significant. 

Instructor's Form , Instructors in the feedback and no-feedback groups 
also completed an ''Instructor ' s Form" at midsemester (Appendix F), This form 
elicited instructor self-ratings on 21 of the 23 SIR items; each of the items 
was reworded slightly for this purpose. 

Instructor's Cover Sheet , Instructors in all three groups also completed 
a "cover sheet 11 which contained several questions dealing with the course and 
the instructor (Appendix G), The number of years of teaching experience and 
the field in which the subject most appropriately belonged were obtained in 
this way. The instructor's sex, also a variable of interest, was ascertained 
from the instructor's name or, when necessary, by inquiring a-*-, the college. 
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Analyses 

Multivariate analysis of variance (MANOVA) and regression analyses were 
the major methods of analyses, In analyzing the possible effects of subject 
area of the course, sex of the instructor, and number of years of teaching 
experience, the sanple was combined across colleges in order to insure adequate 
cell frequencies. Differences between the feedback, no-feedback, and posttest 
groups were also analyzed within each college to examine the possibility that 
teacher changes may vary by college. 

Results and Discussion 

MANOVA results appear in TabDes 2 through 5- In the first three analyses, 
presented in Tables 2, 3, and U, end-of-semester item scores were compared 
among the feedback (treatment) group, the no-feedback (control) group, and the 
posttest group. The multivariate F ratios presented in each table provide a 
test or the differences among the three groups. Also presented in each table 
are the F ratios for differences in ratings due to other factors (e.g., years 
of teaching, sex, subject area, etc.) and, most importantly, the interaction 
or each of these factors with group effects. 

Because of the large number of dependent variables (items), the MANOVA 
analyses were done in two stages. First, the 15 items that were thought to 
have the best chance of reflecting instructional changes were analyzed. Then, 
using those 15 items as covariates, +v e remaining items were analyzed. 
Because the SIR items are not independent, using the 15 items as covariates 
served to minimize their efieet on the succeeding analysis. The second group 



of items consisted of the remaining eight repeated items plus, for two of t he 
analyses, two items from the end-of-semester form dealing with the overall 
effectiveness of the instructor and the overall value of the course to students 
The overall ratings, it was reasoned, might reflect improvement in instruction 
even though the more specific items did not. Specifically, student feedback 
could influence teachers to change in ways that improve their overall effective 
ness but not in the specific ways indicated by the 23 items. 

Years of teaching experience and subject area of the course were included 
in *-he first MANOVA analysis (Table 2) because they were presumed to be par- 
ticularly critical variables. That is, less experienced teichers or those in 
certain subject areas might more likely be expected to be influenced by student 
feedback. Teachers were separated into the three categories of experience: 
those in their first or second year of teaching, those who had been teaching 
three to six years, and those who had taught for seven or more years. The 
subject area divisions were natural sciences, social sciences, humanities, and 
education and applied subjects. 

As the results in Table 2 indicate, there were no differences among the 
three groups nor were there differences in any of the interactions of subject 
area, years of teachxng, and groups. There were, however, differences in the 
end-of-semester ratings given to teachers in various subjecc areas (F ratios 
of 2.k0 and 2.U1, p< .001) and, to a lesser extent, for teachers with varying 
numbers of years in teaching (F = 1.91, P < .01). These findings will be 
discussed later in greater detail. 

In light of the differences in ratings related to the subject area of the 
course, the second MANOVA analysis included that factor once again, along with 



-13- 



Table 2 

Summary of Results of Multivariate Analysis of Variance (MANOVA) 
for Treatment, Number of Years Teaching, and Subject Area 

(N = h35 classes) 



For 15 items 


df hypothesis 


df error 


F 


P< 

It 


icai d J. ccL^il J.ug 


50 


752 


1.24 


.171* 


dudj ecu Area 




1117 . 8 


2.40 


.001 


uroupb v ireaumenu ) 


30 


752 • 


• 83 


.731 


icdib learning x oUDjecu Area 


90 


2121.1 


1.0U 


.387 


Years Teaching x Groups 


6o 


1U69.9 


.86 


.77* 


Subject Area x Groups 


90 


2121.1 


.96 


.602 


Years x Subject x Groups 


180 


3571.* 


1.0U 


.3U2 


For 10 items, with the 
first 15 as covariates 










Years Teaching 


20 


730 


1.91 


.01 


Subject Area 


30 


1072 


2.U1 


.001 


Groups (Treatment) 


20 


730 


• 79 


.731 


Years Teaching x Subject Area 


6o 


1917. k 


1.10 


.288 


Years Teaching x Groups 


ko 


1385.9 


.82 


.77* 


Subject Area x Groups 


6o 


1917. k 


.93 


.626 


Years x Subject x Groups 


120 


28U7.5 


.87 


.8*1 
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the sex of the instructor. The results are presented in Table 3. Group dif- 
ferences were again insignificant, as were the two-way and three-way interac- 
tions of groups, subject area, and sex* As expected, subject area was highly 
significant (p = U.67 and 2.68, p< .001), as was sex of the instructor 
(F = 2-15, p< .001; F = 2.27, p< -01). 

The fourth variable investigated for its possible interaction with treat- 
ment effects was the college. As the results in Table k indicate, feedback did 
not result in significant instructional changes at any of the five colleges, 
although the faculty ratings across the colleges did differ significantly 
(F = 3.63 and 3.^9, p*^ .001; college differences are discussed later in 
this section, as well as in the second section cf this report). 

The final MAN0VA analysis of the first-semester data employed the mid- 
semester ratings as covariates. Differences in enci-of-semester ratings, 
therefore, were compared for the two groups (feedback and no-feedback) for 
which midsemester ratings were available. Colleges and subject areas were 
also included in this analysis. The results, which appear in Table 5, indicate 
clearly that treatment and interaction effects were once again not significant. 
In other words , the feedback group did not differ from the no-feedback group 
even after controlling for midsemester ratings. 

To summarize the findings to this point, end-of-semester ratings of 
instructors who were given midsemester feedback did not differ, as hypothesized, 
from either the no-feedback or the posttest groups. Moreover, teacher ratings 
for the three groups did not differ when subject area, sex of instructor, 
college, or amount of teaching experience were taken into account. In short, 
subgroups of teachers identified by the above categories did not appear to 
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Table 3 

Summary of MANOVA Results for Treatment, Subject Area, 

and Sex of the Instructor 
(N = U35 classes) 



For IS i terns 


dl n r ypotnesis 


dl error 


r 


P < « 


Sex 


15 


388 


3.1$ 


• 001 


subject Area 


US 


1153* U 


u.67 


• 001 


Group s t Ire a tmen t ) 


30 


776 


1*07 


• 360 


Sex x Subject Area 


U5 


11$ 3 


• 66 


• 958 


Sex x Groups 


30 


776 


.58 


.97 


Subject Area x Groups 


90 


2IOO.6 


1.01 


• U5 


Sex x Subject x Groups 


90 


2188.6 


1.0$ 


.3$ 


For 1C items, with the 
first 15 as covariates 










Sex 


10 


386 


2.27 


.01 


Subject Area 


30 


1133.7 


2.68 


.001 


Groups (Treatment) 


20 


772 


.77 


.757 


Sex x Subject Area 


30 


1133.7 


.98 


.U97 


Sex x Groups 


20 


772 


1.2U 


.205 


Subject Area x Groups 


60 


2027. U 


.91 


-66U 


Sex x Subject x Groups 


60 


2027. U 


1.19 


.15U 
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Tablo h 



Summary of MANOVA Results for Colleges and Treatment 

(N = 1*35 classes) 



For 15 items 


df hypothesis 


df 


F 


. P < 


Colleges 


60 


1587.0 


3.63 


.001 


Groups (Treatment) 


30 


ol2 


1.02 


.hho 


Colleges x Groups 


120 


2902.8 


1.02 


.U28 


For 8 items, with the 
first 1$ as covariates 










Colleges 


32 


1U69.3 


3.U9 


.001 


Groups (Treatment) 


16 


796 


1.21 


.29x 


Colleges x Groups 


6h 


2302 


.86 


.781 
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Table 5 

Summary of MANOVA Results for Treatment, Colleges, and 

Subject Areas Using Midsemester Scores as Covariates 

(N = 29h classes) 



For 15 


items 


df hypothesis 


df error 


F 


P< 


Groups 


(Treatment) 


15 


227 


1.07 


.390 


Groups 


x Colleges 


6o 


888.3 


.98 


.518 


Groups 


x Subject Area 


16 


675.1 


1.12 


.285 



change significantly after receiving feedback. Instructors in their first or 
second year of teaching were considered most likely to change since they 
generally had not yet established rigid teaching habits; but student feedback 
did not result in changes for even this less experienced group. 

While so-called treatment effects (i.e., those related to the feedback 
conditions) were not significant, there were a number of differences that are 
worthwhile noting. As Tables 2-5 indicate, ratings varied considerably by 
subject area, sex of instructor, number of years in teaching, and college. 
The specific ways in which student ratings differed for each of these categories 
are outlined below. 

Subject area . Courses in the natural sciences, relative to those in 
humanities, social sciences, and education and applied areas, were seen by 
students as having a faster pace, as being more difficult, and as being 
less likely to stimulate student interest. In addition, students perceived 
natural science teachers as less open to other viewpoints. 

Humanities teachers also had a few shortcomings according to the student 
ratings: in comparison to teachers in the other three subject fields, they 
were less likely to inform students of how they were to be evaluated, and there 
was less agreement between announced objectives and what was actually taught 
in their courses. 

Generally speaking, students rated courses or teachers in the social 
sciences and those in education and applied areas similarly. (The item mean 
scores for each of the subject areas are listed in Appendix H, ) 

Instructors se x. Students rated women teachers more favorably than men 
teachers on several of the items, particularly those dealing with what is 



frequently referred to as "teacher-student interaction, 11 Women teachers, 
students indicated, were more likely to know when students didn't know the 
material, were more concerned with student progress, made more comments on 
papers or exams, and generally made better use of class time. Courses taught 
by male teachers, on the other hand, were more stimulating to students, although 
they were also viewed as more difficult. 

Number of years of teaching experience . There was only one way in which 
teachers with varying numbers of years of teaching experience differed in their 
ratings: those with more than six years of experience were less likely to 
inform students of how they would be evaluated in the course. This is somewhat 
surprising. The general expectation is that teachers improve with experience. 
However, the more experienced teachers in this study were rated no better than 
these in their first or second years of teaching; in fact, on this one item 
they were rated less favorably. 

College differences . The major difference in the student ratings among 
the five colleges was College Five's poor ratings. On a third of the itejis, 
students rated instructors at College Five less favorably than at the remaining 
four institutions. In particular, students at College Five thought that 
instructors were not concerned enough about students (items 5 and 8) nor did 
they do enough to challenge students (item 10). The differences will be 
discussed more fully in the second part of this report, where they will be 
related to instructor self-ratings. The mean item scores for each of the col- 
leges are listed in Appendix I. 
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Ir addition to the differences noted for College Five, there were also 
some minor variations among the colleges. All of these variations would suggest 
ve.y different learning (or teaching) environments among the institutions, or 
quite possibly different levels of expectations among students , Further studies 
are needed to explain why ratings differ among colleges and to investigate what 
student ratings might contribute toward a better understanding of teaching 
climates at particular colleges, 

[ tudent Feedback and Self-ratings of Teachers 

A major hypothesis of this study was that changes in instruction would be 
related to instructor self-ratings. Specifically, th<. expectation was that 
student feedback would lead to improved instruction for chose teachers who had 
rated themselves much better than their students had rated them. The rela- 
tionship, moreover, was predicted to be linear: the greater the discrepancy, 
the greater the likelihood of improvement. To test this hypothesis, the 
following regression equation was employed with the feedback and no-feedback 
groups : 

R 2 = a x + + c (I - R x ) , 

where is the predicted second-semester rating, R is the midsemester 

rating, I is the teacher self- rating (thus I - R is the difference between 
the instructor self-rating and the midsemester rating), and the a ! s, b ! s, 
and c ! s are the regression weights. If the hypothesis is supported, there 
should be a .significant difference between the regression weights for I - R^ 
(i,e,, c ) for the feedback and no-feedback groups, with c for the feedback 
group being positive and greater. 
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For these analyses , instructors in both groups were divided into those 
who rated themselves more favorably and those who rated themselves less 
favorably than their students rated them on each item. Of particular interest 
were teachers who rated themselves more favorably, since the prediction was 
that student feedback would effect changes only for those teachers. For most 
of the items, about 60-65 per cent of the sample had rated themselves more 
favorably; also, for this group the size of the discrepancy between self and 
student ratings was much greater than for the group that rated themselves 
less favorably, (A further analysis and discussion of instructor-student 
discrepancies in ratings is given in the second part of this report,) 

Results of the regression analyses *°or 17 of the 19 agree-disagree items 
appear in Table 6 (instructors did not respond to items 10 and 20 because they 
were not appropriate as self-rating items; also, the first four items were not 
scored appropriately for this analysis). Listed are regression weights c and 
t test results of the difference in c between the feedback and no-feedback 
groups. These results are presented for instructors who had rated themselves 
less favorably (left side of the table) and more favorably (right side) than 
their students rated them. Results for instructors who rrted themselves less 
favorably indicate fairly random differences in c , and on only one item 
did the feedback and no-feedback groups differ. 

But differences in c for teachers who rated themselves more favorably 
were significant (p < ,05) for 5 of the 17 items, as indicated in the last 
column. Equally important is the fact that for 13 of the 17 items, the direc- 
tion of the differences also supports the hypothesis. That is, for those items 
the c f s for the feedback group (group 1 in the table) are higher than those 
for the no-feedback group (group 2), 
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Table 6 

Summary of Results of Regression Analyses 



For instructors who For instructors who 

rated themselves less rated themselves more 

favorably 4 favorably e 



Item 


Group 0 


Regression 
Weight c 


* f 
difference 


Regression 
Weight c 


t 

f 

difference 


5 


1 
2 


-.077 
-.OOF 


-.1*07 


.189 
.0*45 


1.23 


6 


1 

2 


.120 
.129 


-.0356 


.026 
.167 


-1.21* 


7 


1 

2 


-.23** 
.137 


-1.97 


.310 
.168 


1.11 


8 


1 

2 


.079 
.01*9 


.2*48 


.091 
-.033 


.738 


9 


1 

2 


.007 
.026 


-.182 


.010 
.033 


-.232 


11 


1 

2 


.202 
.083 


.1*1*3 


.11*2 
-.077 


2.1*66 


12 


1 

2 


see footnote b 


.167 
.217 


-.368 


13 


1 
2 


.097 
-.001 


.1*51 


-.01*3 
-.075 


.1*81* 


Hi 


1 
2 


.117 
.176 


-.1*11 


-.070 


1.1*1* 


15 


1 

2 


.228 
.25*4 


-.067 


.235 
-.150 


? 196* 


16 


1 

2 


.077 
.130 


-.363 


-.120 

.01*9 


-.826 


IT 


1 

2 


-.118 
-.002 


-.625 


.101* 

.003 


.883 


18 


1 

2 


.079 
.015 


.607 


.172 
-.031 


2.062* 


19 


1 

2 


-.0*42 
.132 


-1.337 


.130 
-.155 


3.932 


21 


1 

2 


-.005 
.087 


-1.171* 


.111 
-.093 


2.618* 


22 


1 

2 


-.035 
-.21*0 


.831 


.121* 

.001* 


1.21 


23 


1 

2 


-.098 
.061 


-1.^5 


.190 
.002 


1.1*5 



Significant at the .05 level, 
^or the formula: = & 1 + b ]L R ]L + c (I - R ) 

b Item numbers refer to the midsemester form. Instructors did not respond 
to numbers 10 and 20. For item 12, all instructor responses for one of the 
groups were identical; thus c could not be computed . v 

0 

1 = Feedback group, 2 = No-feedback group. 

d Less favorably defined as I - R greater than 0. 

e More favorably defined as I - R = less than 0. 
f 

Test of the difference in regression weights c for (I - R ) in 
groups 1 and 2. 1 
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The five items for which there were signif ica: t differences in regression 
weights c for instructors who rated themselves more favorably were: 
(11) The instructor encourages students to think for themselves 
(15) In this class I feel free to ask questions or express my opinions 

(18) The instructor summarizes or emphasizes major points in lectures 
or discussions * 

(19) My interest in the subject area is being stimulated by this course 
(?l) I have been putting a good deal of effort into this course. 

For ibese items in particular, then, the findings suggest that instructors 
changed after receiving feedback if_ their own ratings were especially better 
than their students 1 ratings — that is, if instructors were "unrealistic" in 
how they viewed their teaching. Even with only a half semester left, instruc- 
tors apparently were able to stimulate student interest, eifort, and thinking; 
to becoiLe more open to questions or opinions; and to do a better job of sum- 
marizing or emphasizing major points. 

Changes I 1 Instruction Following A Longer Time Period 

Thus far, \,he results indicate that changes in instruction at the end of a 
single semester were rather limited. Only those teachers who rated themselves 
much better than did their students appear to have changed after receiving mid- 
semester feedback. It may be, however, that more teachers do in fact change, 
but not until the following semester. Perhaps teachers need more time to think 
about and develop new practices, and perhaps they find changes easier to make 
with the start o** a new course. 

To investigate this possibility, additional data were collected at the end 
of the Spring semester at one of the five colleges. The particular college 



was one at which 30 teachers had been randomly selected to use the SIR rating 
form at the later time rather than during the Fall semester. In short, they 
were to be the Spring control group . In addition to this group, teachers in 
the Fall feedback and posttest groups at this same college were asked: (l) if, 
p during the Spring semester, they were teaching the same course in which they 

1 used the SIR form in during the Fall, and (2) if they would be willing to 

administer the form in that course at the end of the Spring semester (they 
' were surveyed about one month prior to the end of the semester) . Eight 

teachers from the feedback group and 13 from the posttest group responded 
affirmatively to both questions and did administer the form once again at the 
end of the Spring semester. Although a larger sample would have beer desir- 
able, the teacher ratings were mean scores and thus more reliable th^n indi- 
vidual scores. In sum, the eight teachers in the feedback group were using 
the form for the third time, while the posttest group, which had administered 
the form only at the end of the Fall semester, were using it for the second 
time. Of the 30 instructors asked to use the SIR form for the first time, 
21 were able to do so. A summary of the Spring groups, including the number 
in each group and the specific times when SIR had been used by the groups, 
is given in Figure 1. 

Time of Administrations 

Midsemester, End oi* Semester., End of Semester, 
Group Fall Fall Sp ring 

Fall Feedback Group, N = 8 X X X 

Fall Posttest Group, N = 13 X X 

Spring Group 5 N = 21 X 

Figure 1 

Time of Administrations of SIR for each 
of the Three Spring Groups 
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While the multivariate analysis of variance of the Fall data did not reveal 
significant differences between the feedback and comparison groups, there were 
ei^ht items for which the univariate F values for one or more of the analyses 
had approached significance (i.e., p between ,05 and .20). On the basis of 
this prior finding, and because they would appear to be most sensitive to 
change, these eight items rather than the entire set of 23 were selected for 
further analysis with the Spring groups. To test the differences among the 
three groups on these eight items at the end of the Spring semester, multi- 
variate analysis of variance was used once again. 

The MANOVA results clearly indicated that the groups differed (F = 2.l8; 
df = l6,26U; p< .015). An inspection of the univariate F values and the means 
for each group, presented in Table 7, further indicated that for most of the 
items, the feedback groups received more favorable scores than either of the 
other two groups (with the exception of item l6, lower scores are more favoraole). 

In particular the feedback group differed from the posttest and Spring- 
only groups in that they more likely summarized or emphasized major points in 
lectures or discussions (item lU ) , were better prepared for class (12), were 
more likely to accomplish course objectives (20), made better use of class time 
(3), and finally, according to their students, more likely made helpful comments 
on papers or exams (9). On the only other item on which the groups differed, 
the Spring-only group received the most favorable score; that was on item 19, 
"the instructor was open to other viewpoints. 11 

These results suggest then that student feedback did effect some changes 
in instruction over time, in that teachers who had received feedback twice 
during the previous semester did receive better ratings than instructors who 
had received feedback once or not at all. Before jumping to that conclusion, 
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however, some alternative explanations need to be considered. Perhaps teachers 
in the feedback group who chose tc readminister the items again in the Spring 
were better to begin with. Indeed, they may have chosen to readminister the 
rating items because they had already received favorable ratings and sought 
further reinforcement for their efforts. This potential bias, to some extent, 
was minimized because only courses that were taught in both the Fall §nd 
Spring semesters were included in the Spring feedback and posttest groups. Yet 
the possibility of prior differences existed, and to investigate that possibility, 
differences in scores on each of the eight selected items at the end of the Fall 
were tested for three pairs of groups: 

(1) the 8 teachers from the feedback group vs. 35 from the same group 
who did not participate in the Spring; 

(2) the 13 teachers from the posttest group vs. 32 from the same group 
who did not participate in the Spring; and 

(3) the 8 teachers in the feedback group vs. the 13 in the posttest group. 
If none of these differed significantly in the Fall, then a strong case 

could be made for the Spring differences being due to student feedback. MANOVA 
tests for the three pairs of comparisons did not in fact yield significant 
differences. End-of-fall ratings for the eight teachers from the feedback 
group, therefore, were not unlike their colleagues' ratings in that group, 
nor were they significantly different from the 13 in the posttest group; like- 
wise, end-of-fall ratings for the 13 in the posttest group were similar to 
the other faculty ratings from that sample. Differences at the end of the 
Spring term, consequently, were less likely to be due to prior differences or 
self-selection. It would seem safe to conclude that student feedback did 
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effect changes in the feedback group, and that these changes were reflected 
in certain Spring ratings. 

But what about the Fall posttest group? Because their Spring ratings 
were very similar to the group that used the form for the first time, it 
would appear that the single feedback had little effect in changing instruction. 
There are several possible explanations for this lack of change- The posttest 
group, unlike the feedback group, had not received any comparative or "norma- 
tive" information to help them interpret their scores. It may be, therefore, 
that the lack of interpretive information did not enable those instructors to 
understand fully their ratings (particularly since student ratings are typically 
skewed in a positive direction); consequently, they may not have thought they 
needed to change. Or it may be that again not enough time had lapsed for 
changes to be made (one semester vs. a semester and a half for the feedback 
group). Or finally, perhaps at least two sets of student ratings are needed 
before teachers see a pattern of weaknesses that they might improve. 

Summary and Conclusions 

The purpose of this study was to investigate the effects of feedback 
from student ratings on changing instructional practices at the college level. 
Teachers within each of five diverse colleges were assigned randomly to a 
feedback (treatment), no-feedback (control), or posttest group. The feedback 
and no-feedback groups used a 23-item student rating form in one of their 
classes at midsemester during the Fall 1971, and also responded to an "instruc- 
tor's Form" with self-ratings of instruction. A summary of their students 1 
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responses along with some comparative information to aid in interpretation 
were sent back to each instructor in the feedback group within a week, while 
results were withheld from the no- feedback group. Both groups used the rating 
form (the Student Instructional Report) in the same class again at the end of 
the semester; the posttest group, which was using it for the first time, also 
administered the form in one class at the end of the Fall semester. 

If student feedback improved instruction, end-of-semester ratings of the 
feedback group should have been better than either the no-feedback or the 
posttest groups. Multivariate analysis of variance results for the end-of- 
semester ratings, however, indicated no significant differences among the 
three groups. Furthermore, no differences were noted when various interactions 
were investigated. Specifically, regardless of the particular college, subject 
area of the course, sex of the instructor, or the number of years the instructor 
had taught, the feedback did not appear to produce a difference. (There were, 
however, some noteworthy differences in student ratings of instructors by 
college, subject area, sex, and years of experience.) 

But a major hypothesis of this study was that student feedback would 
effect changes in teachers who had rated themselves more favorably than their 
students had rated them. Results of the regression analyses indicated this 
to be the case; the findings suggested that the greater the discrepancy — where 
the discrepancy reflected the extent to which students rated teachers less 
favorably than the teachers apparently expected — the greater the likelihood 
of change. On the basis of equilibrium theory, teachers were predicted to 
change because they valued student opinion and because students had evaluated 
them less fave^ably than they had expected. In order to restore a condition 
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of "equilibrium," teachers who received such feedback changed in the direction 
suggested by the student ratings. 

It would appear, tht^., that student ratings can effect changes in teaching 
practices even after only a half-semester. Those changes, however, were evi- 
denced only for those teachers who had unrealistica]?y high (compared to their 
students 1 views) opinions of their instructional practices. Others may have 
rated themselves average or poorly, just as cheir own students rated them, and 
did not change even though there was room for improvement. 

But would more teachers change if they had more time— that is, if more 
than a half-semester had lapsed? It appeared that they vould, especially if 
they are provided with information to help them interpret their own results. 
A segment of the teachers in the Fall sample u;ed the rating form again in 
the Spring. The feedback group, which had at that point administered the form 
for the third time, scored better on several items than a random group of 
instructors who were using it for the first time, or the so-called Fall post- 
test group, which had used it once before but did not receive any comparative 
information to aid in interpretation ♦ The feedback group, therefore, not only 
had the benefit of more time to change (a semester and a half) but they also 
were the only group that had been given interpretative information. This 
information consisted of simply the mean scores "based on some 80 teachers who 
had used the form during the previous year* 

The need for comparative or "normative 11 data to help the teacher under- 
stand better his or her students 1 rating is, therefore, underscored by this 
study. Comparisons with other teachers at their institution, with instructors 
from similar colleges, or with a national sample of teachers in the same 
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subject area would undoubtedly help teachers recognize strengths and weaknesses 
otherwise overlooked. Other interpretative mechanisms, such as providing each 
teacher with a narrative summary of their student ratings along with specific 
suggestions for change, or providing individual or group counseling to teachers 
with poorer ratings, could also add to the understanding and use of student 
rating results. 

This study was designed to show what student feedback coupled with some 
rather minimal comparative information could do toward changing instruction. 
The "treatment" in this study purposely paralleled what is typically done with 
student ratings at most colleges: the results were seen only by the instructor 
and, while some comparative information was provided, interpretation of the 
results as well as what to do about them was left up to the instructor. Some 
might view the changes that occurred in this study as modest. But in view of 
the ease with which student ratings can be employed for instructor self- improve- 
ment — indeed the exten ; to which they are already in use among colleges and 
universities— they appear to have had a sufficient impact to warrant § continued 
use as one method of improving college teaching. 
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II. SELF-RATINGS OF COLLEGE TEACHERS: A COMPARISON WITH STUDENT RATINGS 



Teacher self-ratings have been proposed as a possible source of 
information for performance improvement and, to a lesser extent, as an 
input into performance evaluation. As a basis for decisions on promotion 
or salary, self-evaluations are not likely to have much validity. But 
it is possible that some form of systematic self-evaluation could be 
helpful to the teacher trying to improve instruction, particularly if 
combined with external evaluations provided by students or colleagues* 

There has been little research on teacher self-ratings. In particular, 
the relationship between self-ratings and those provided by students or 
colleagues is not yet fully known. With 51 instructors in a military 
setting, Webb and Nolan (1955) reported a correlation of .62 between 
instructor self-ratings and student ratings. Clark and Blackburn (l97l) 5 
however, reported a correlation of .19 between student ratings and faculty 
self-ratings at a small college, and a similarly moderate correlation (.28) 
between self-ratings and colleague ratings. In both of these studies, 
overall teaching was rated rather than specific instructional practices. 

The purpose of this study was to further investigate college teachers 1 
self-ratings and ratings given by students by comparing these two sets of 
ratings over a wide range of specific, student-oriented instructional 
practices. Discrepancies between self-ratings (or self-descriptions) and 
those provided by students would underscore the need foj student feedback 
to the instructor as well as highlight specific areas of instruction where 
feedback is most essential. Differences in ratings will also be studied 
to investigate their relationships to selected teacher and course characteristics. 

ERIC 
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Procedure 

The sample for the study consisted of 3^3 teaching faculty at five 
institutions of higher education. Between 75 to 90 per cent of the teachers 
invited from each college participated in the study. The five institutions 
included two state colleges (one of which had a predominantly black enroll- 
ment), a selective liberal arts college, a multipurpose college, and an 
urban community college. None of these institutions had, at the time of 
the study, a systematic program to collect student ratings, nor did a 
significant portion of t.ieir faculty collect student ratings on their own. 
The majority of teachers in this study, therefore, ;:ere not familiar with 
how students might rate their instruction. 

Students and teachers responded to 21 items dealing with instructional 
practices. The student questionnaire was titled the "Mi ds ernes ter Student 
Instructional Report" and actually contained 23 items, 21 of which were 
judged appropriate for instructor self-ratings. Included were items that 
faculty members in an earlier study had identified as providing information 
they would like to receive from students (Centra, 1972). Among the 
dimensions of instruction included were the organization of the course, 
student-teacher interaction, instructor communication, student effort, and 
stimulation of students. Previous factor analytic studies had identified 
several of these as dimensions that effectively differentiated among 
instructors (Cofftnan, 195I1; Gibb, 1955; Hodgson, 1958; Isaacson, McKeachie, 
Milholland, Lin, Ho feller, Baerwalt, & Zinn, 196U). 

Responses to 17 of the items were on a four-point agree-disagree scale, 
with a "not applicable" option also provided. The four remaining items 
used a four- or five-point scale wi\h different response options for each 

ERJC 



-35- 

item. The wording for each of the statements in the questionnaire 
differed slightly for students and instructors. For example, an item on 
course objectives was worded as follows for each group: 

For students: The instructors objectives have been made clear 

For teachers: I feel my objectives for the course have been made 
clear to students 

Teachers were asked to "describe this course, your teaching, or the 
students enrolled." They were told that the reason for obtaining this 
self-report was to see which items were tapping information already known 
to most instructors. 

The data were collected at midsemester of the Fall 1971 term. Instruc- 
tors administered the rating form in one class of their own choosing, with 
the understanding that only they would receive a summary of their students 1 
responses . 

Analyses 

Faculty-student comparisons were made in a number of ways. First, 
the relationship between the two sets of ratings was studied by correlating 
instructor responses to each of the 21 items with the mean responses of 
students in their class (N = 3^3 classes). Secondly, differences between 
the way faculty as a group and students as a group rated or described 
instruction were investigated by a comparison of means; i.e., the mean 
score for all teachers on each item was compared to the average of the 
student class means. 

Finally, the discrepancy between each instructor's response and the 
mean response of his class was of particular interest. The extent of that 
discrepancy and its relationship with specific teacher or course variables 
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(i.e., sex, years of teaching experience, subject area of the course) 
tfere analyzed through multivariate analysis of variance. 

Results and Discussion 

The results of the comparison of means and the correlational analysis 
for items 5-21 are presented in Table 1. The correlation between the two 
sets of descriptions or ratings was not particularly high, indicating only 
modest agreement in the way faculty and students perceived instruction. 
While the correlation between faculty and student responses was significantly 
different from zero for most of the items due to the large N (3^3), the 
median correlation was only .21. 

Also listed in Table 1 are the mean faculty responses for each item 
and a ranking of the items, the mean of the classroom (student) means and 
a ranking of those scores, the results of the t-tests, and the number of 
colleges where the difference between the means was significant. A 
graphical presentation of the data is presented in Figure 1. Responses 
for items 5-21 could range from one for "strongly agree" to four for 
"strongly disagree"; thus, lower values represent greater agreement with 
each statement. The comparisons of the mean values indicate that instructors 
as a * oup generally rated or described their teaching more favorably than 
did their students . (Students 1 ratings were also skewed toward the more 
favorable end of the scale, which is usually v ? case with this vype of 
instrument.) In particular, instructors and students did not agree on the 
following items: the extent to which students are free to ask questions 
or give opinions in class (item lU) 5 the extent to which instructors are 
concerned with student learning (13), the amount of agreement between 
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Fig. 1. Faculty and student mean responses to items in in- 
structional report. 
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objectives and what is being taught (6), instructor openness to other 
viewpoints (20), the extent to which instructors inform students of how 
they would be evaluated (l6), whether the instructor encourages students 
to think for themselves (10), and the clarity of course objectives (5). 
For each of these seven items, instructor-student differences were 
notable at either four or all five of the colleges. 

On the other hand, there was little difference between the faculty 
and student groups in their ratings of the instructor preparation for 
class (15) and on the extent to which course objectives were being accom- 
plished (21 )• For the remaining eight items, the differences were modest 
and in many instances not significant within a college . 

Another vay to look at the data is to compare items with each other. 
The question then becomes: To what extent do the groups of teachers and 
students order the items similarly? A ranking of item means for each of the 
two groups indicates fairly high similarity; in fact, a rank correlation 
of .11. This would suggest that, while teachers and students are generally 
using different points on the scale in responding to the items (as 
indicated by the comparison of means), both groups tend to see the same 
relative strengths and weaknesses among the teachers in this study. For 
example, while there is a large mean difference between the groups on 

instructor concern wiih student lea ng (item 11 ), both groups rated 
instructors favorably on this item ^ 1 comparison to other aspects of 
teaching . Keeping in mind that higher scores represent unfavorable 
(disagree) responses, botfr groups also rated the instructors in this study 
poorly on stimulating student interest in the course (l8). 
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Generally speaking, combining the ranks of both teachers and students 
indicates that not stimulating student interest enough (l6), the lack 
of helpful comments on papers or exams (12), and not knowing when students 
understand the material tended to be rated as the most frequent criticisms 
of instruction for the teachers in this study. On the other hand, their 
strengths were in allowing students to feel free to ask questions or give 
opinions (ih) and in their concern with student learning (11). 

Individual Teacher-Class Differences 

Probably more important than a comparison of the way an average 
instructor and an average class rated instruction is some knowledge of how 
many instructors perceived themselves far differently than their students 
did. A distribution of the differences between each instructor's 
responses and those of his class (i.e., the class means) provides that 
information. Presented in Table 2 is a summary of the results of such a 
distribution. For each item, the percentage of instructors who gave them- 
selves "considerably poorer" or "considerably better" ratings is indicated 
within each college and for the total sample. A difference of .63 or 
greater was used to define "considerably poorer or better" because a 
difference of at least that great would appear to be large enough to 
have some practical significance; it is also the approximate standard 
deviation for most of the student item responses. 

For most of the items, between a fourth and a third of the instructors 
described or rated themselves considerably better than their students did. 
The median, in fact, was Just under 30 per cent for all 3^3 instructors 
and their classes. Forty-one per cent of the instructors gave themselves 
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better ratings on item ik : students are free to ask questions or give 
opinions in class; and 36 per cent on item 11: the instructor is concerned 
about whether students learn and tries to be actively helpful. Both items 
deal with faculty-student interaction as do items 8, 9, 10, and l6 for 
which fairly high percentages of instructors also gave themselves better 
ratings. The faculty-student interaction dimension, then, appears to be 
one on which a sizable number of instructors and their students do not 
agree and on which student reactions would appear to be especially crucial. 
Other similar areas would be the instructor's openness to other viewpoints 
(item 20) and the agreement between announced objectives for the course 
and what was being taught (6). 

A surprisingly large percentage of instructors rated themselves poorer 
than students did in a few areas. Fifteen per cent rated themselves more 
poorly on class preparation and 12 per cent were less satisfied that they 
were accomplishing course objectives. In general, however, only between 
h to 8 per cent of the teachers gave themselves considerably poorer ratings. 

One of the items in the form was unique in that it elicited opinions 
on student effort in the course (19). For students, the exact wording was: 
"I have been putting a good deal of effort into this course"; for instructors 
it was worded: "Students seem to be putting a good deal of effort into this 
course." The results for this item, as one might expect, were much different 
than those for other items. Compared to students 1 responses, 18 per cent 
of the faculty thought students generally were putting considerably less 
effort into the course, while 10 per cent gave students better ratings on 
effort than students gave themselves. In other words, in this instance 
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students have tended to give themselves better ratings just as instructors 
did on so many of the previous items. 

An inspection of the differences within each college indicates fairly 
similar results with the exception of college five. In comparison to the 
other four colleges, higher percentages of the instructors at college 
five rated themselves considerably better than did their students on a 
majority of the items. While it is not possible to conclude much on the 
basis of one college, it is interesting to note that college five was the 
smallest and most selective of the colleges in the study. Moreover, in- 
structors at college five were given the poorest student ratings among the 
five colleges, whereas their self-ratings were not much different or poorer 
than those of instructors elsewhere. Thus, the gap between instructor- 
student ratings at college five was due largely to the poorer ratings by 
students, perhaps because of higher expectations on their part, rather 
than on better ratings by instructors. 

Presented in Table 3 is a summary of responses to the first four items, 
which used varied responses rather than agree-disagree options. The items 
deal with the pace, the level of difficulty, and the work load of the 
course, as well as the extent to which the instructor used examples and 
illustrations. Once again there were student-instructor differences although 
they were not particularly large. Instructors tended to think they more 
often used examples and illustrations, and at three of the colleges 
instructors more likely considered the pace at which material was covered 
to be slow. College five, the selective liberal arts college, was once 
again noteworthy in that its faculty and to some extent the students 
reported less frequent use of examples or illustrations in courses. 



L Pace at which material 
is covered: 



Table 3 

Faculty-Student Comparisons at Five Colleges and Total (N = 3^3), 
for Four Items in Instructional Report Questionnaire 

a 

Percentage Responding 
Student s Faculty 
College College 
1 2 3 k 5 Total 12 3 k 5 Total 



Very or somewhat slow 
Very or somewhat fast 



9 10 7 8 6 9 22 2k 10 8 Ik 16 
26 20 27 23 33 25 20 28 2k 30 30 27 



Level of difficulty of 
course for students 
enrolled: 



Very or somewhat elementary 
Very or somewhat difficult 



11 13 10 10 9 11 
31 25 32 21 38 30 



10 7 10 k 8 7 
26 31 37 37 hi 3k 



Work load of course rela- 
tive to others: 

Lighter 
Heavier 



18 22 17 1° 18 19 
20 21 27 29 27 25 



25 2k 21 17 Ik 20 
35 23 32 32 33 30 



Extent to which examples 
and illustrations were 
used: 



Frequently 


60 


70 


76 


67 


58 


67 


88 


75 


86 


82 


65 


80 


Occasionally 


28 


26 


20 


26 


3U 


26 


12 


21 


lit 


18 


32 


19 


Seldom 


10 


It 


It 


6 


8 


6 


0 


2 


0 


0 


3 


1 


Never 


2 


1 


1 


1 


1 


1 


0 


2 


0 


0 


0 


1 



For- items 1-3, the four responses have been collapsed into two categories; the 
middle response ("about right" or "about the same") is not shown. 
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A final question regarding individual teacher-class differences was 
whether those differences were related to instructors of different sexes, 
with varying amounts of teaching experiences, or those teaching different 
subject areas. Are the self-ratings for female teachers, for example, 
more similar to their students' ratings than are those of male teachers? 
For this analysis, each course was grouped int,;> one of four general subject 
area categories: natural sciences, humanities, social sciences, and 
education and applied subjects (e.g., business, home economics, nursing). 
Teaching experience consisted of three categories: one or two years, three 
to six years, and seven years or more, Data for 235 teachers were available 
for this analysis. 

The results of the multivariate analysis of variance, in which all 
21 items were used as variables, are given in Table k. There were no 
differences due to sex or years of teaching experience or for any of the 
interactions; there was, however, a significant difference (p < .05) 
due to subject area. This difference was largely beween natural science 
courses and those in education and applied subjects. Specifically, teachers 
in the natural sciences did not think th'3 pace of the course was as fast 
as their students said it was, and they did not think students put as much 
effort into the course as students said they did. Conversely, teachers 
in education and applied subjects reported the course as having a faster 
pace than their students reported, and thought that students put more 
effort into the course than students said they did. 
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Table U 

Summary of MANOVA Results of Instructor-Class Differences 
by Sex, Subject Area, and Number of Years Teaching 

(N = 235) 





df 


df 






Source 3, 


Hypothesis 


Error 


F 


P < 


Sex 


21 


192 


.3** 


.99 


Years of Teaching 


k2 


38!i 


1.09 


.3h 


Subject Area 


63 


57U 


1.33 


.05 


Sex x Years Teaching 


k2 


38U 


.86 


.72 


Sex x Subject Area 


63 


5lh 


.62 


.99 


Years Teaching x Subject Area 


126 


1121 


.85 


.89 



'The triple-order interaction was not run because one of the cells was blank. 
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Summary and Conclusions 

A comparison of students' ratings of instruction with teachers 1 self- 
reported ratings in over 300 classes at five colleges disclosed a modest 
relationship between the two sets of evaluations. The median correlation 
for IT items was .21, indicating that faculty members generally evaluate 
or describe their teaching somewhat differently from the way it is 
evaluated or described by their students. Not surprisingly, the highest 
correlations occurred for the more factual items, on which there was some- 
what less chance for disagreement (e.g., the instructor informs students 
of how they would be evaluated) while items eliciting opinions (e.g., 
the instructor is using class time well) resulted in the lowest correlations. 

As mentioned earlier, previous studies, in which students and faculty 
ratings of instruction had been compared, employed a single overall measure 
of teaching and produced conflicting results: .62 in one instance (Webb & 
Nolan, 1955) and .19 in the other (Clark & Blackburn, 1971). The latter 
correlation was reported for college teachers and, of course, was fairly simil 
to the median correlation for the 17 items used in the five-college study 
reported here. Webb and Nolan's use of instructors in a military setting 
may explain the unusually high correlation found in their study; in any 
event, it does not seem to apply to more typical college teaching situations. 

In addition to the general lack of agreement between self and student 
evaluations, there was also a tendency for teachers as a group to give 
themselves better ratings than their students did. In a sense this 
tendency might be viewed as only "human," or certainly not surprising. 
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As Robert Burns has reminded us, most people do not see themselves as others 
see them; teachers and the way they see their instruction are apparently 
no exception. 

Comparisons between student and faculty responses were also made 
across items, and a rank correlation of .77 indicated a good deal of 
similarity in the way the two groups rank ordered the items. This suggests 
that instructors are indeed aware of many of their particular teaching 
strengths and weaknesses, even though they see themselves more favorably 
in absolute terms. They are also probably more aware of their own relative 
strengths and weaknesses than they are of the way they might compare to 
other instructors, as suggested by the previously cited correlational data 
for ea<;h item. An ipsative approach to student rating of faculty, there- 
fore, ,n which the emphasis is on identifying the specific "good" and "bad" 
practices of each individual teacher, would not appear to be as informative 
to instructors as the normative approach, in which comparisons may be made 
with ocher relevant groups of instructors. 

The discrepancy between individual teacher ratings and the mean rating 
given by his class was most notable for between a fourth to a third of the 
3^3 instructors in the study, and in particular for items related to student- 
instructor interaction, course objectives, and the instructor's openness 
to other viewpoints. These areas of instruction, then, would seem to be 
particular ones in which a sizable proportion of teachers could profit from 
student feedback. 
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Teacher-student discrepancies were about the same for men and women 
teachers and for the more and less experienced teachers. That there were 
no sex differences in rating discrepancies is not particularly surprising; 
but one might have predicted that the self-ratings of more experienced 
teachers would be closer to student ratings. Since most of the teachers 
in this study had not made a practice of obtaining systematic feedback 
from their students, the findings suggest that getting to know student 
reactions to teaching is not something that comes merely with experience. 

Of particular interest, however, were differential discrepancies 
noted for the subject areas; teachers of natural science subjects under- 
estimated (relative to their students) both the pace of their course and 
their students 1 efforts, while teachers of education and applied subjects 
overestimated the course pace and their students 1 efforts. These subject 
area differences might be explained by the differences in the content and 
in the intended objectives of courses in each area. Instructors of 
mathematics, physics, biology, and the like may feel that there is so much 
factual and theoretical material to cover in their courses that a fast 
pace coupled with a good deal of student effort is a necessity. What 
teachers in the natural sciences view as an acceptable pace and work load, 
however, apparently doea not coincide with their students, who frequently 
are using courses in other fields for comparison. In education and applied 
subject areas, not only might the amount of factual material be less 
demanding on students, but frequently the major objectives of the courses 
are to establish particular attitudes or skills with students. Working 
toward those objectives may result in courses that appear slower paced to 
students , 
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In conclusion the results of this study would argue for the collection 
of student ratings as a means of providing instructors with information 
they do not already have about their teaching. As an aid to instructional 
improvement, teacher self-ratings might in fact be used in conjunction with 
student feedback as a means of highlighting discrepancies for the individual 
instructor. 
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Appendix A 

Examples of Letters Sent to 
Colleges and Individual Teachers 
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Initial letter tc some 60 colleges* 
Self-addressed postcard was also included* 



EDUCATIONAL TKSTING SERVICE 



PRINCETON, N.J. 0854 



Area Code 609 
921-9000 
CABLE-EDUCTESTSVC 

Developmental Research Division 



We have received a grant to study the effects of student ratings of 
instruction. More precisely, we are interested in assessing the impact of 
formal student feedback on teaching behavior. The study, to be conducted 
next fall, will provide participating colleges with the free use and scoring 
of a rating form that we have been developing over the past year. Question- 
naire responses will be returned only to the individual instructor. 

In order to invite colleges to participate we first need to obtain some 
general information about the extent to which student rating forms are used 
on college campuses. Could you or a staff member please respond to the three 
brief questions pertaining to your institution listed on the enclosed self- 
addressed postcard? We would appreciate your reply by May 5>. Your responses 
do not commit you to participate in the study; if invited, you will of course 
be provided with more details. 

Thank you. 

Sincerely, 



John A. Centra m 
Research Psychologist 
Project Director 
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Invitation to colleges to participate, 
KIH'C'ATIONAI. TESTING SERVICE IMUNCKTO.V .N.J. 0854G 



At at Code 009 
921 . 9000 
t.\MK-MH C'ttlS iSlL 
• 

Devcl off turn la / Rnra i ch Division 



A short while ago you received a letter briefly describing a study we will 
be conducting during the fall, 1971 semester dealing with student ratings of 
instruction. We would like to invite College to participate in this 

study. As mentioned in the earlier letter, your faculty will be asked to use 
a student rating form that we have been developing here at ETS over the past 
year. Student responses to the questionnaire will be scored and returned to 
each professor at no cost to the college. 

The major purpose of the study is to investigate the effectiveness of 
formal student reactions in modifying college instruction. Funded by the 
Esso Education Foundation, the study will hopefully shed some light on whether 
student ratings lead to positive changes in instruction. The procedure is as 
follows : 



1, We would select a portion of your faculty to use a brief 
rating form at midsemestei in one of their classes. The form will 
take only 10-15 minutes for students to complete. Half of this group 
of instructors will receive a summary of student responses a few days 
later; the other half will not receive these results until after the 
end of the semester. 



2, At the end of the semester we would like the remaining faculty 
as well as the original group to administer our rating form in one of 
their classes. For that portion of the faculty that used the form at 
midsemester, this will represent a second use in the same class; other 
faculty will be employing it for the first time. The second rating 
form will be identical to the first except for some additional questions 
pertinent to an end of course appraisal. Once again the questionnaires 
will be scored at ETS and sent back to each instructor. 

Thus one portion of your faculty will use a student rating form twice 
during the semester and we will compare end-of-semester responses for those 
who received their results immediately vs, those who did not. The third 
group of instructors will simply use the form at the end of the semester 
which is normally the case. Further analyses with all three groups will 
also help us learn more about the particular items included in the form. 
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We plan to include seven or eight colleges in this study and we hope that 
can be one of these. The information that individual faculty 
members receive from the rating form should be of interest to them. Further- 
more, summary results will be made available for each college, which will 
provide you with an overall view of student reactions to instruction at 

If you feel your college ce.n participate, we will need to hear 
from you as soon as possible. We would also need the name of a person at the 
college who would act as our liaison. 

If you need further information about the study at this time, feel free 
to call me collect at (609) 921-9000, extension 2793. 

I will look forward to hearing from you. 

Sincerely, 



John A. Centra 
Research Psychologist 
Project Director 
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KDl'CATIONAI. TKSTINC5 SERVICK PRINCETON. N.J, 085 K) 



Area Code 609 

921-9000 . October Uu 1971 

i. 4 RLE- E I) ( Y;7 ES 'I Ml 
Developmental Rehear elf Division 



Dr. , Vice President for Academic Affairs at , mentioned to you 

in his memo of September 20 that College along with several other colleges 

was participating with us in a project dealing with student course evaluation. 
The project, funded by the Esso Educational Foundation, is investigating what 
students are able to evaluate in the classroom and how useful this information 
might be to the individual instructor. 

Your participation in the project will be brief and, hopefully, of value 
to you. There are 30 copies of a mid-semester Student Instructional Report 
enclosed. These should be used in one of your classes--of your own choosing— 
which will be meeting on October 18, 19, or 20. If you need additional copies 
of the questionnaire, they are available in Dr. office. After the forms 

have been administered, put them back in the enclosed envelope (which has my 
name on it), seal it (to maintain confidentiality) and put it in interoffice 
mail. The mail clerk will set them aside for me. They should be returned no 
later than Wednesday the 20th. 

The questionnaires wilWbe scored here at ETS, summarized, and returned to 
you as voon as possible. Only you will receive this summary of student responses. 
Because of the number of colleges and faculty members participating in this 
project and possible delays in processing the forms, we may not be able to get 
all of the summaries back at the same time. 

We will be also asking you to administer an end-of -semester form in the 
same class. That form is slightly longer than the mid-semester form and allows 
you to add questions of your own for scoring. 

There is one last request. One of the questionnaires is marked 11 Instructor 1 s 
Form." Would you respond to the items according to what you feel you deserve-- 
that is, according to how you would describe or rate yourself . While this may 
be difficult to do for a few of the items, we have good reason to ask for the 
information: we would like to know the extent to which some of the items may be 
tapping information that most instructors are already aware of. The question- 
naire with your responses should be enclosed along with those filled out by 
students. 



Thank you for your cooperation. You will receive the end-of -semester 
questionnaire around December 1. 



ERJ£ '/ John A. Centra 



Sincerely., J 

/ John A. Centra 
i/ Research Psychologist 



EDUCATION AL TESTING SERVICE 



PRINCETON, N.J. 08540 



Area Code 609 
921-9000 
CABLE. F.DL'CIESTS VC 



Developmental Research Division 



October 1U, 1971 



Dr. , Vice President for Academic Aff airs j at , mentioned to you 

in his memo of September 20 that College along with several other colleges 

was participating with us in a project dealing with student course evaluation. 
The project, funded by the Esso Educational Foundation, is investigating what 
students are able to evaluate in the classroom and how useful this information 
might be to the individual instructor. 

Your participation in the project will be brief and hopefully of value to 
you. I would like you to have your students complete an "Instructional Report" 
questionnaire in one of your classes at the end of the fall semester. This 
questionnaire consists of 39 items which elicit student perceptions and ratings 
of the course and instruction; it should take students only 10-15 minutes to 
complete. The form also allows you to add questions of your own for scoring. 
Student responses will be scored, summarized by us, and sent back to each 
instructor. Only the individual instructor will receive these summaries. 

You will receive a supply of questionnaires in your mailbox along with 
additional instructions around December 1. 



Thank you for your assistance. I hope you will find your participation 
worthwhile . 





John A. Centra 
Research Psychologist and 
Project Director 
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EDI'CATIONAL TESTI.NT. SERVICE PRINCETON, N.J. 085-10 



A tea Code 609 - , , 

m-9000 October 1U, 1971 

CABLF.-EDUCTESTSVC 
Developmental Research Division 



The enclosed material deals with the project on student ratings of instruc- 
tion which Dean has mentioned to you and which I described briefly at 
the first faculty meeting. The project, you may recall, is investigating what 
students are able to evaluate in the classroom and how useful this information 
might be to the individual instructor. 

There are 30 copies of a mid-semester Student Instructional Report enclosed. 
These should be used in one of your classes which will be meeting on October 18, 
19, or 20. If you need additional copies of the questionnaire, they may be 
obtained in Dean office. After the forms have been administered, put 

them back in the envelope in which they were received, seal it (to maintain con- 
fidentiality) and return it to one of the mailboxes in the General Office which 
has been reserved for that purpose. You should return the forms no later than 
Wednesday the 20th. 

The questionnaires will be scored here at ETS, summarized, and returned to 
you as soon as possible. Only you will receive this summary of student responses. 
Because of the number of colleges and faculty members participating in this 
project and possible delays in processing the forms, we may not be able to get 
all of the summaries back at the same time. 

As you may recall, we will be also asking you to administer an end-of -semester 
form in the same class. That form is slightly longer than the mid-semester form 
and allows you to add questions of your own for scoring. 

There is one last request. One of the questionnaires is marked "Instructor's 
Form. 11 Would you respond to the items according to what you feel you deserve — 
that is, according to how you would describe or rate yourself . While this may 
be difficult to do for a few of the items, we have good reason to ask for the 
information: we would like to know the extent to which some of the items may be 
tapping information that most instructors are already aware. The questionnaire 
with your responses should be enclosed along with those filled out by students. 

Thank you for your cooperation. You will receive the end-of -semester 
questionnaire around December 9» 





John A. Centra 
Research Psychologist and 
q Project Director 



ERIC 



-59- 



Letter to feedback group, sent with midsemester class results. 



EDUCATIONAL TESTINO SERVICE 



PRINCETON. N.J. 08540 



A tea Cvdr609 
921-9000 
CA RLE-ED UCTES TSVC 



Developmental Research Division 



To the Instructor 



Enclosed is an IBM "printout" summarizing your students 1 responses to the 
mid-semester Student Instructional Report , Your name, course identification, 
and the number of students responding should appear at the top. 

The printout shows the percentage of students who chose various options to 
each question and the average (mean) response to each. A standard deviation is 
also provided for those of you who would like some indication of variation in 
responses. 

Also enclosed is a response summary based on 2960 students in 75> classes. 
This information was collected for 17 of the 23 items at one four-year institu- 
tion last Spring. It may help you interpret your own responses, although it 
is by no means offered as ideal normative data. It is, however, all that is 
now available. 

We hope you will find the information provided you at this time useful. 
As mentioned in previous correspondence, only you have received these results. 
The end -of -semester form, which is slightly longer and provides you with the 
opportunity to add questions of your own for scoring, will be sent to you in 
early January, 

Thank you for participating in the project thus far. 




JAC:mh 
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KDITATIONAI. TKSTINCi SKKVICK 



I* R I N C" KTO.N, N.J. 08540 



Arm Code 609 
921-9000 
CABlxK-ktM MEM SIC 



November 29, 1971 



Developmental Research Diviuoti 



To the Instructor 



To complete the second and final phase of our project dealing with student 
ratings of instruction and to give you the opportunity to obtain additional 
feedback from your students, you will find enclosed some end-of -semester Student 
Instructional Report forms. The form contains 39 items plus a section for up 
to 10 supplementary questions that you may add** 

The questionnaire should be administered in the same class and section in 
which you administered the mid-semester forms ♦ That is imperative. I have con- 
sequently enclosed the same number of questionnaires that you used at mid-semes- 
ter (plus a few extra). You should administer the forms during any class period 
of this last week of the semester. After they have been administered, put them 
back in the envelope in which they were received and seal it to maintain confiden- 
tiality* Then you should cross your name off the address label, add mine, and 
put it in interoffice mail where the mail clerk will set them aside for me (if 
you prefer you can drop it off personally to the mail clerk) . 

You should return them on or before the last day of class (December 8). 
We will score them and send you a summary of responses as soon as possible. 
Once again only you will receive that summary. 

Also enclosed is a green-printed form titled Instructor's Cover Sheet . 
We have already written or coded much of the necessary information on the left 
side of the sheet, including your course, but we need some additional information 
from you for questions A through I. They will take only a minute to conplete. 
The cover sheet should be returned in the same envelope as the forms filled out 
by students. It is important that you complete and return this cover sheet so 
that we can identify your class. It is also important that you direct your stu- 
dents to write the ETS report number at the top of their answer sheet. That 
five digit number is written at the top of your cover sheet. 

Thank you once again for your help with this project. I hope you will find 
your results helpful* A report of this five college study will be available 
next Spring; let me know if you would like a copy. 




John A. Centra 
Research Psychologist 
Project Director 



Make sure students respond to both sides of the questionnaire. 
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KDl'CATIONAI. TKSTINCJ SEKVICE 



Area Code 609 
921 . 9000 
l.tBIE-hlilCIFMSVC 



PKINCKTON, N.J. 08540 



December 3, 1971 



Developmental Refenreh DwiiU.n 



To the Instructor 

To complete the second and final phase of our project dealing with student 
ratings of instruction and to give you the opportunity to obtain additional 
feedback from your students, you will find enclosed some end -of -semester Student 
Instructio nal Report forms . The form contains 39 items plus a section (I V) for 
student responses to up to 10 supplementary questions that you may add. 

The questionnaire should be administered in the same class and section in 
which you administered the mid-semester forms . That is imperative. I have 
consequently enclosed the same number of questionnaires that you used at mid- 
semester (plus a few extra) . You should administer the forms during any class 
period before December 17. After they have been administered, put them back 
m the enclosed envelope and seal it to maintain confidentiality. Then return 
the envelope to Mr. office (Institutional Research). 

You should return them to Mr, before the Christmas holiday. 

We will score them and send you a summary of responses as soon as possible. 
Once again only you will receive that summary. 

Also enclosed is a green -printed form titled Instructor's Cover Sheet. 
We have already written or coded much of the necessary information on the left 
side of the sheet, including your course designation, but we need some additional 
information from you for questions A through I. They will take only a minute to 
complete. The cover sheet should be returned in the same envelope as the forms 
filled out by students. It is important that you complete and return this cover 
sheet so that we can identify your class. It is also important that you direct 
your students to write the ETS Report Number ,at the top of their answer sheet. 
That five-digit number is written at the top of your cover sheet. 

Thank you once again for your help with this project. I hope you will 
find your results helpful. A report of tnis five-college study will be avail- 
able late next Spring; let me know if you would like a copy. 

Sincerely, 

/ 



/ John A. Centra 
/ Research Psychologist 
Project Director 
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KI>1 CATIO S'AI, TKSTINC; SKKVICK 



PRINCETON. N.J. 08540 



Ami Code 609 
921*9000 
LiHIE-UH C'l t:vi\\ ( 



December 3, 1971 



Dcvtl oftmcnta I Re«n t c/i Dit •/ won 



You will undoubtedly recall that you administered a Student Instructional 
Re P° rt in one of your classes at mid-semester as part of our project dealing 
with the effectiveness of student ratings of instruction. Enclosed is a com- 
puter printout summarizing your students 1 responses. 

Although I could not say so at the time you were initially contacted, it 
was necessary to withhold these responses from you (and many of your faculty 
colleagues) in order to carry out the project. Briefly, we are trying to find 
out in what way, if any, student feedback may modify instructional procedures. 
Part of the faculty at were given a summary of student responses 

a few d^ys after administering the forms, and the other part, which included you, 
were not given that information until now. Both faculty groups were determined 
on a strictly random basis . 

To conplete the project and to give you the opportunity to obtain additional 
information from your students, I've enclosed a batch of end-of -semester Student 
Instructio nal Report forms. The form contains 39 items plus a section for stu- 
dent responses to up to 10 supplementary questions that you may add (Section IV) . 

The forms should be administered in the same class and section in which you 
administered the mid-semester forms. That is imperative. Consequently I've 
enclosed the same number of questionnaires that you used at mid-semester (plus 
a few extra). You should administer the questionnaire during any class period 
before December 1?. After they have been administered, put them back in the 
enclosed envelope and seal it to maintain confidentiality. Then return the 
envelope to Mr. office (Institutional Research). 

Ion should return the forms to Mr. before the Christmas holiday. 

We will score them and send you a summary of responses as soon as possible. 
Only you will receive that summary. 

Finally, you will find enclosed a green-printed form titled Instructor's 
Cover Sheet . We have already written or coded much of the necessary information 
on the left side of the sheet, including your course designation, but we need 
your responses to questions A through I. They will take only a minute to com- 
plete. The cover sheet should be enclosed in the same envelope as the student 
forms* It is inportant that you complete and return this cover sheet since it 
is the only way that we can identify the information for your course. It is 
also iinportant that you instruct your students to write the ETS Report Number 
at the appropriate place on their answer sheet. That five-digit number is written 
toward the top of your cover sheet. 



ERIC 
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Thank you once again for your help on this project. If you would like 
additional details about it, I would be happy to provide them. A report of 
this five-college study will be available late next Spring; let me know if 
you would like a copy. 



fiincerejly, 

/" 




John A. Centra 
• 1 Research Psychologist 
Project Director 
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KDl'CATIONAL TESTING SERVICE 



PRINCETON, N. J. 08540 



A 'fa Codt 609 
921*9000 
< UUE-h.IH CIES'ISM 



December 3, 1971 




To the Instructor 



You will recall receiving a letter from me in October indicating that you 
would receive a supply of questionnaires from me at this time dealing with our 
project on student ratings of instruction. The project is funded by the Esso 
Educational Foundation and is investigating what students are able to evaluate 
in the classroom and how useful this information might be to the individual 
instructor. 

Enclosed are 35 copies of the Student Instructional Report which I would 
ask you to use in one of your classes (of your own choosing). If you need addi- 
tional forms, Mr. has a limited supply. You should administer 
th^ forms during any class period before December 17. After they have been admin- 
istered, put them back in the enclosed envelope and seal it to maintain confiden- 
tiality. Then return the envelope to Mr. office (institutional 
Research) • 

You should return them to Mr. before the Christmas holiday. We 
will score them and send you a summary of responses as soon _ possible. Only 
you will receive that summary . 

You will note that the form contains 39 items plus a section (IV) for 
student responses to up to 10 supplementary questions that you may add. Make 
sure students respond tc both sides of the questionnaire. 

Also enclosed is a green-printed form titled Instructor 1 s Cover Sheet . 
We have already written or coded much of the necessary information on the left 
sid\ of the sheet, but we need some additional information from you for ques- 
tions A through I. They will take only a minute to complete. The cover sheet 
should be returned in the same envelope as the forms filled out by students. 
It if ^ important that you complete and return this cover sheet so that we can 
identify your class. It is also important that you direct your students to 
write the ETS Report Number at the top of their answer sheet. That five-digit 
number is written at the top of your cover sheet. 

Thank you for your help on this project. I hope you will find your 
results helpful. A report of this five-college study will be available late 
next Spring j let me know if you would like a copy. 




ERIC 
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Appendix C 

Midsemester and End-of -Semester 
Student Instructional Report 
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Midsemester 
STUDENT INSTRUCTIONAL REPORT 



Name of Instructor 



Name of Course 



This questionnaire gives you an opportunity to express anonymously your perceptions of this course 
and the way it has been taught Responses wiJl be summarized and given only to your instructor. 



SECTION I. Items 14. 



Directions Circle one response number for each question. Use any pen or 
pencil you have handy. 



For me, the pace at which the instructor 
covers the material is. 



3. 



The work load for this course in relation 
to other courses of equal credit is 



Very slow 
Somewhat slow 
Just about right 
Somewhat fast 
Very fast 



Much lighter 
Lighter 

About the <amc 
Heavier 
Much heavier 



For my preparation and ability, the level 
of uifficulty of this course is: 



4. 



Very elementary 
Somewhat elementary 
About right 
Somewhat difficult 
Very difficult 



To what extent docs the instructor use 
examples or illustrations to help clarify 
the material? 

1 Frequently 

2 Occasionally 

3 Seldom 

4 Never 



SECTION II. Items 5-23. Directions' Circle the number that represents the response closest to your 
opinion. 

NA (0) = Not Applicable or don 7 know. The statement does not apply to this course or 
instructor, or you simply are not able to give a knowledgeable response 

Strongly Agree. You strongly agree with the statement as it applies to this 
course or instructor. 



SA (1) 

A (2) 

0 (3) 

SD (4) 



Agree. You agree more than you disagree with the statement as it applies to this 
course or instructor. 

Disagree. You disagree more than you agree with the statement as it applies to 
this course or instructor. 

Strongiy Disagree . You strongly disagree with the statement as it applies to this 
course or instructor. 







NA SA 


A 


D 


SD 


5. 




. 0 1 


2 


3 


4 


6. 


There has been considerable agreement between the announced objectives of the 














0 1 


2 


3 


4 


7. 


The instructor is using class time well 


0 1 


2 


3 


4 


8. 


V i instructor is readily available for consultation with students 


0 1 


2 


3 


4 


9. 


TK instructor seems to know when students don't understand the material 


0 1 


2 


3 


4 


10. 




0 1 


2 


3 


4 


11. 




0 1 


2 


3 


4 


12. 


The instructor seems genuinely concerned about whether students learn and is 














0 1 


2 


3 


4 


13. 




0 1 


2 


3 


4 


14. 




0 1 


2 


3 


4 


15. 


In this class I feel free to ask questions or express my opinions 


0 1 


2 


3 


4 


16. 


The instructor is well-prepared for each class 


0 1 


2 


3 


4 


17, 


The instructor has informed students of how they would be evaluated in the course 


0 1 


2 


3 


4 


18. 


The instructor summarizes or emphasizes major points in lectures or discussions . . 


0 1 


2 


3 


4 


19. 


My interest in the subject area is being stimulated by this course 


0 1 


2 


3 


4 


20. 


The scope of the course is too limited; not enough material is being covered .... 


0 1 


2 


3 


4 


21. 




0 1 


2 


3 


4 


22. 




0 1 


2 


3 


4 


23. 




0 1 


2 


3 


4 



THANK YOU 



Copyright (c) 1971 by Educational Testing Service. All rights reserved. 



_ Student Instructional Report 

W INSTITUTIONAL RESEARCH PROGRAM FOR HIGHER EDUCATION 



This questionnaire gives you an opportunity to express anonymously your views of this course 
and the way it has been taught. Indicate the response closest to your view by blackening the 
appropriate oval. Use a soft lead pencil (preferably N 2) for all responses to the questionnaire. 
Do not use an ink or ball point pen. 



ETS Report Number 



SECTION I Items 1-20. Blacken one response number for each question. 



NA (0) 



SA 


(1) 


A 


(2) 


D 


(3) 


SO 


(4) 



= Not Applicable or don't know . The statement does not 
apply to this course or instructor, or you simply are not 
able to give a knowledgeable response. 

= Strongly Agree . You strongly agree with the statement 
as it applies to this course or instructor. 



ment as it applies to this course or instructor. 
Disagree. You disagree more than you agree with the 
statement as it applies to this course or instructor. 
Strongly Disagree. You strongly disagree with the 
statement as it applies to this course or instructor. 



1. 
2. 

3. 
4. 
5. 
6. 
7. 
8. 

9. 
10. 
11. 
12. 
13. 
14. 

15. 
16. 
17. 
18. 
19. 
20. 



The instructor's objectives for the course have been made clear , . x , 

There was considerable agreement between the announced objectives of the course and 

what was actually taught , .. 

The instructor used class time well 

The instructor was readily available for consultation with students 

The instructor seemed to know when students didn't understand the material..:. 

Lectures were too repetitive of what was in the textbook(s) \ 

The instructor encouraged students to think for themselves 

The instructor seemed genuinely concerned with students' progress and was actively 

helpful , .... ...... 

The instructor made helpful comments on papers or exams 

The instructor raised challenging questions or problems for discussidn 

In this class I felt free to ask questions or express my opinions 

The instructor was well-prepared for each class , 

The instructor told students how they would be evaluated in the course 

The instructor summarized or emphasized major points in lectures or discussions 



My interest in the subject area has been stimulated by this course 

The scope of the course has been too limited; not enough material has been covered. 

Examinations reflected the important aspects of the course 

I have been putting a good deal of effort into this course , 

The instructor was open to other viewpoints 

In my opinion, the instructor has accomplished (is accomplishing) his objectives 
for the course 



NA 



SA 



(5 



• & ± 



it 

. A 



D SO 



o 

i 



SECTION II Items 21-31. Blacken one response number for each question. 



21. For my preparation and ability, the 
level of difficulty of this course was: 

O Very elementary Somewhat difficult 

Somewhat elementary cr Very difficult 
a About right 

22. The work load for this course in relation 
to other courses of equal credit was: 

Much lighter co Heavier 

>2 Lighter x Much heavier 

a About the same 



23. For me, the pace at which the instructor 
covered the material during the term was: 

o Very slow 
c£ Somewhat slow 
a> Just about right 



<£> Somewhat fast 
a Very fast 



24. To what extent did the instructor use examples 
or illustrations to help clarify the material? 

O Frequently a> Seldom 

<£> Occasionally <s> Never 



Copyright f> 1971 by Educational Testing Service. All Rights Reserved. 
Pn « «e i *» M.J. 0nfi40 
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25 Was class size satisfactory for the 28. What grade do you expect to receive in 

method of conducting the class? this course? 

Yes, most of the time a No, class was too small c A t Fail 

No, class was too large "C It didn't make any differ- a B t Pass 

ence one way or the other C No credit 

Which one of the following best X D £ Other 

describes this course for you? 

Major requirement or i College requirement but 29 - Wnat is vour approximate cumulative 

elective within major field not part of my major grade-point average? 

Minor requirement or or minor field C 3.50-4.00 <S> 1.00 1.49 

required elective out- of Elective not required in 1 3.00-3.49 t Less than 1.00 

side major field anyway a> 2.50-2.99 None yet- fresh men 

. r3; Other q> 2.00-2.49 or transfer 

?? ? Which one of the following was your most :r 1.50-1.99 

I important reason for selecting this course? 

i rnend(s) recommended it 30. What is your class level? 

j ' Faculty advisor's recommendation c Freshman i Senior 

j Teacher's excellent reputation a> Sophomore <r Graduate 

j Thought I could make a good grade o> Junior <e other 

! Could use pass/no credit option 

j it .vas required 31. Sex: 

Suoject was of interest q Female 

| Other a Male 

j 

jSECTION til Items 32-39. Blacken one response number ~ ^ " 

1 for each question. J? v . 0< 

i *S </ </ / * <t 

5 32. Overall, I would rate the textbook(s) q a , Q: a ; ^ 

33. Overall, I would rate the supplementary readings r& C £> <x j ,v & 

;34. Overall,! would rate the quality of the exams , a <z> <x t <r 

;35. I would rate the general quality of the lectures ^ e a ^ i <v 

<36. I would r-ate the overall value of class discussions ^ r - ^ a , m < 

37. Overall, I would rate the laboratories ^ rr ^ - " " 

^8. I would rate the overall value of this course to me as & & ^ Gy . 

'39. Compared to other instructors you have had (secondary school and college), how effective 
has the instructor been in this course? (Blacken one response number.) 

One of the most More effective Not as effective One of the least 

. effective than most About as most effective 

(among the top 1 0%) (among the top 30%) average (in the lowest 30%) (in the lowest 1 0%) 

i SECTION IV Items 40 49. If the instructor provided supplementary questions and response options, use 

! this section for responding. Blacken only one response number for each auestion 
: NA NA 

.40. : r - - '5 '/ t t 45, <v <x d a> a? x a, <? 't r? 

0 ,} if - '* J r > # 46. <s> <t> a> a> ^) <$> <s> -?„ r r 

4 2. ft 2 3 t <» *, 1 47i <fc * O) fl& ^ (J) * 7 dy y 

43. ' »* j l ,v fJ# vi - ^ <J; 43 <r> o> ^ ^ ^ ^ ^ ^ 

j44. 'J. " ^ °' f 2 ^ ,J ^ ^ 49! ^ ^ Cb & cc .% 

I SECTION V Students' Comment Section 

If you would like to make additional comments about the course or instruction, use a separate 
sheet of paper. You might elaborate on the particular aspects you liked most as well as those 
you liked least. Also, how can the course or the way it was taught be improved? PLEASE 
GIVE THESE COMMENTS TO THE INSTRUCTOR. 



If you have any comments or suggestions about this questionnaire (for example, the content or 
responses available), please send them to: Student Instructional Report, Educational Testing 
Service, Princeton, New Jersey 08540, 
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Appendix D 

Reliability of the Student Instructional Report Items 1 
Item Number (SIR 

End-of -Semester Fonn) Estimated reliability for the following 

number of individuals in each class 





5 


10 


15 


20 


25 


35 


1 

J. 


.39 


•56 


.66 


• 72 


• 76 


.82 


o 

c 


.37 


• 5k 


.6u 


.71 


.75 


.81 


3 


.50 


.67 


r-i r-> 

• 75 


.80 


.83 


.88 


1, 


• U9 


.66 


.7U 


.79 


.83 


.87 




.56 


.72 


.80 


.81* 


.87 


.90 


o 


.50 


.66 


• 75 


.80 


.83 


.87 


< 




.6u 


• 72 


.78 


.81 


.86 


8 




• 0£ 


.0 ( 


OA 
.7O 


• 92 


.9k 


9 


.39 


.56 


.66 


.72 


.76 


.82 


10 


.37 


.5U 


.61* 


•71 


.75 


.81 


11 


.50 


.67 


.75 


.80 


.83 


.88 


12 


.1*9 


.66 


.71* 


.79 


.83 


.87 


13 


.56 


.72 


.80 


.81* 


.87 


.90 


11* 


.50 


.66 


.75 


.80 


.83 


.87 


15 


.1*7 


.61* 


.72 


.78 


.81 


.86 


16 


.69 


.82 


.87 


.90 


.92 


.91* 


17 


.71 


.83 


.88 


.91 


.93 


.95 


18 


.57 


.73 


.80 


.81* 


.87 


.90 


19 


.52 


.68 


.76 


.81 


.81* 


.88 


20 


.58 


.71* 


.81 


.85 


.88 


.91 


21 


• 31 


.1*7 


.57 


.61* 


.69 


.76 


22 


.1*5 


.63 


.71 


.77 


.81 


.85 


23 


.57 


.72 


.80 


.81* 


.87 


.90 


21* 


.28 


• 1*U 


.51* 


.61 


.66 


.73 


32 


.66 


.79 


.85 


.88 


.91 


.93 


31* 


.59 


.71* 


.81 


.85 


.88 


.91 


35 


.58 


.71* 


.81 


.85 


.87 


.91 


36 


.59 


.71* 


.81 


.85 


.88 


.91 


38 


.55 


.71 


.79 


.83 


.86 


.90 


39 


.65 


.78 


.85 


.88 


.90 


.93 



Based on analysis of variance (see Winer, 1962, pp. 12U-132). 
Fifteen students were randomly selected from each of 28 classes 
for this analysis. Estimated reliabilities for the various num- 
bers of individuals were calculated by Spearman-Brown. 
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Appendix E 

Midsemester-End of Semester Correlations 

for Twenty- Three SIR Items 1 

Item Number Feedback No-Feedback 

(mid- semester form) Group 1 Group 2 

N = 137 teachers N = 1$9 teachers 

1 .70 .77 

2 -73 .7$ 

3 .83 • .80 
It .71* 71 

5 .66 .72 

6 .58 .68 

7 .78 .76 

8 .61 .68 

9 .6k .69 

10 .76 .6$ 

11 .73 .7$ 

12 .71 .72 

13 .67 .67 
lit .79 .79 

15 .77 . 7li 

16 .80 .69 

17 .55 .73 

18 .71 .69 

19 .79 .72 

20 .6$ .60 

21 .72 .6U 

22 .73 .67 

23 .69 ■ .71 



These correlations could be viewed as test-retest 
reliability. 



Appendix F 

INSTRUCTOR'S FORM -71_ 

(To be completed by the instructor) 

Please respond to each item according to how jroH would describe this course, your teaching, 
or the students enrolled. The items parallel those in the student form and will gi/e us 
the opportunity to see if some of the items are tapping information already known to most 
instructors. 

Your Name Name of Course : 

SECTION I. Items 1-u. Directions ; Circle one response number for each question. Use 

any pen or pencil you have handy. 

1. For the students enrolled, the pace 3. In my opinion the work load for this 
at which the material in this course course in relation to other courses 
is being covered is: is probably: 

1 Very slow 1 Much lighter 

2 Somewhat slow 2 Lighter 

3 Just about right 3 About the same 
1* Somewhat fast [j Heavier 

$ Very fast 5 Much heavier 

2. For the students enrolled, the level U. The extent to which I have been using 
of difficulty of this course is: examples and illustrations to help 

1 Very elementary clarify the material of this course is: 

2 Somewhat elementary 1 Frequently 

3 About right 2 Occasionally 
1* Somewhat difficult 3 Seldom 

5 Very difficult ^ Never 

SECTION II. Items 5-21. Directions : Circle the number that represents the response 

closest to your opinion. 

NA (0) = Not Applicable or don't know . The statement does not apply to this course or 

your teaching, or you simply are not able to give a knowledgeable response . 
SA = Strongly Agree. You strongly agree with the statement as it applies to this 

course or your teaching. 
A (2) ■ Agree. You agree more than you disagree with the statement as it applies to 

this course or your teaching. 
D (3) = Disagree. You disagree more than you agree with the statement as it applies 

to this course or your teaching. 
SD = Strongly Disagree. You strongly disagree with the statement as it applies 

to this course or your teaching. 

w NA SA A D SD 

|. I feel my objectives for the course have been made clear to students.. 0 12 3k 

6. There has been considerable agreement between the announced objectives 

of the course and what is being taught 0 1 2 3 1* 

7. I feel that I have been using class time well * 0 1 2 3 1* 

8. I have been readily available for consultation with students 0 1 2 3 1* 

9. I feel I know when students don 1 !, understand the material 0 1 2 3 1* 

10. I encourage students to think for themselves in this course 0 1 2 3 1* 

11. I have been genuinely concerned about whether students learn and 

try to be actively helpful 0 12 3 1* 

12. I make a point of adding helpful comments on students papers or exams... 0 12 3 1* 

13. I have been raising challenging questions or problems for discussion.. 0 12 3 1* 
11*. In this class, students are free to ask questions or express their 

opinions # # q ^ 2 3 1* 

15. I think that I have been well-prepared for each class 0 1 2 3 1* 

16. I have informed students of how they would be evaluated in the course. 0 12 3 1* 

17. I have summarized or emphasized major points of lectures or discuss ions . . 0 12 3 1* 

18. I **eel that students* interest in the subject area is being stimulated 

by this course 0 12 3k 

^ 19. .Students seem to be putting a good deal of effort into this course 0 1 2 3 h 

ERIC 2 ° # feel that 1 ^ ° pen ^° other viewpoints 0 1 2 3 1* 

Haa^a 21. I feel that I am accomplishing my objectives for the course at this point . , 0 12 3 1* 
« wutKK von ■ - - 
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INSTRUCTOR'S COVER SHEET 

STUDENT INSTRUCTIONAL REPORT 
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Studenb Instructional Report Summary Data 

Mean of Mean Student Responses for Each of Five General 
Subject Areas and for the Total (N=a36 Instructors) 

See questionnaire for wording of items and responses. Note: For most items 
a lower mean value is a better (i.e., more positive) response. ' 



ERIC 



Item 


Biological 
Sciences 
Teachers 
N=29 


Humanities 
N=la6 


Math, Physical 
Sciences 
Engineering 
N=67 


Social 
Sciences 
N=73 


£s<auis a. UJ.UIJ cuJU 

Applied Areas 
N=121 


Total 
N=a36 


1 


1.67 


1.69 


1.70 


1.70 


1.63 


1.68 


2 


1.72 


1.78 


1.71 


. 1.78 


1.71 


1.7a 


3 


1.66 


1.70 


1.6a 


1.73 


1.71 


1.70 


It 


1.62 


1.60 


1.61 


1.57 


1.58 


1.59 


5 


2.09 ' 


1.93 


2. Oh 


1.9a 


1.88 


1.9a 


6 


2.9a 


3.01 


3.00 


3.02* 


2.98 


3.00 


7 


1.87 


1.70 


1.79 


1.75 


I.63 


1.71 


8 


1.82 


1.75 


1.80 


1.7a 


1.72 


1.75 


9 


2.2^ 


1.90 


2.09 


2.12 


1.99 


2.01 


10 


2. Oh 


1.85 


1.93 


1.86 


1.76 


1.85 


11 


1.77 


1.69 


1.7a 


1.62 


1.55 


1.65 


12 


l.l»8 


1.51 


1.50 


1.5U 


1.57 


1.53 


13 


1.66 


1.81t 


1.80 


1.66 


1.65 


1.7a 


lh 


1.69 


1.69 


1.70 


1.66 


1.67 


1.68 


15 


2.01 


2.02 


2.20 


1.92 


1.8a 


1.98 


16 


3.27 


3.19 


3.25 


■ 3.19 


3.13 


3-19 


17 


2.0a 


2.07 


1.88 


2.01 


2.1a 


2.05 


18 


1.76 


1.92 


I.89 


1.93 


1.77 


1.86 


19 


1.80 


1.78 


1.85 


1.71 


1.71 


1.76 


20 


1.6a 


I.69 


1.75 


1.68 


1.6a 


1.63 


21 


3.53 


3.27 


3.61 


3.28 


3.18 


3.32 


22 


3.a3 


3.11 


3.30 


3.12 


3.13 


3-17 


23 


3.a6 


3.18 


3.37 


3.16 


3.11 


3.20 


2h 


1.38 


l.a9 


1.31 


l.aO 


1.39 


1.1a 


32 


2.31 


2. al 


2.81 


2.U9 


2.38 


2.hl 


33 


2.52 


2.m 


2.86 


2.a5 


2.h0 


2.51 


3a 


2.55 


2.U9 


2.U8 


2.h9 


2.U6 


2.U9 


35 


2.21 


2.09 


2.12 


2.17 


2.10 


2.12 


36 


2.63 


2.39 


2.55 


2.h2 


2.19 


2.38 


37 


2.75 


2.6a 


2.37 


2.57 


2.13 


2.kk 


38 


2.36 


2.22 


2.U6 


2.19 


2.0a 


2.21 


39 


2.23 


2.15 


2.20 


2.15 


2.08 


2.U4 



Appendix I 

Student Instructional Report Summary Data -Tit- 
Mean of Mean Student Responses for Each of Five Colleges 
and for the Total (N=436 Instructors) 

See questionnaire for wording of items and responses. Note: For most items, 
a lov/er mean value is a better (i.e., more positive) response. 



Item 


College 1 
N=69 


College 2 
N=113 


College 3 
N=65 


College 4 
N=137_ 


College 5 
N=52 


Total 
N=436 


1 


1.67 


1.53 


1.66 


1.74 


1.88 


1.68 


2 


1.80 


1.58 


1.75 


1.79 


1.87 


1.74 


3 


1.67 


1.58 


1.62 


1.77 


1.88 


1.70 


4 


1.64 


I.46 


1.63 


1.61 


1.70 


1.59 


5 


1.97 


1.78 


1.92 


2.01 


2.12 


1.94 


6 


2.87 


3.11 


2.94 


3.02 


2.97 


3.00 


7 


1.75 


1-58 


1.70 


1.75 


1.88 


1.71 


8 


1.76 


1.57 


1.74 


1.86 


1.88 


1.75 


9 


1.92 


1.86 


1.94 


2.14 


2.25 


2.01 


10 


1.83 


1.71 


1.84 


1.87 


2.15 


1.85 


11 


I.65 


1.52 


1.66 


1.68 


1.86 


I.65 


12 


1.61 


1.43 


1.53 


1.55 


1.58 


1.53 


13 


1.71 


1.59 


1.73 


1.77 


2.03 


1.74 


14 


I.63 


1.57' 


1.70 


1.74 


1.81 


1.68 


15 


2.01 


1.74 


2.02 


2.07 


2.20 


1.98 


16 


• 3.09 


3.21 


3.13 


3.24 


3.22 


3.19 


17 


1.96 


2.06 


2.03 


2.08 


2.10 


2.05 


18 


1.89 


1.73 


1.90 


1.89 


1.99 


1.86 


19 


1.74 


1.60 


1.81 


1.82 


1.91 


1.76 


20 


1.73 


1.50 


I.65 


1.75 


1.85 


1.68 


21 


3.23 


3.21 


3.35 


3.39 


3.44 


3.32 


22 


3.01 


3.09 


3.18 


3.27 


3.26 


3.17 


23 


3.15 


3.12 


3.22 


3.26 


3.29 


3.20 


24 


I.46 


1.34 


1.39 


1.43 


1.50 


1.41 


32 


2.29 


2.26 


2.61 


2.64 


2.56 


2.47 


33 


2.41 


2.27 


2.64 


2.68 


2.62 


2.51 


34 


2.33 


2.30 


2.49 


2.65 


2.64 


2.49 


35 


2.08 


1.88 


2.09 


2.25 


2.37 


2.12 


36 


2.24 


2.09 


2.26 


2.51 


2.99 


2.38 


37 


2.52 


2.16 


2.51 


2.54 


2.87 


2.44 


38 


2.18 


1.94 


2.20 


2.32 


2.55 


2.21 


39 


2.18 


1.95 


2.02 


2.26 


2.35 


2.14 



